A de-duplication environment can include multiple actors. For example, two different computers may participate in a de-duplication action. A first computer may de-duplicate an object and communicate it to a second computer that will store and index the object. Conventionally, if two computers wanted to work together on an action like a de-duplication action, then a central control would dictate conditions and parameters under which the two computers would communicate and/or co-operate. As the number of communicating and co-operating computers has grown, this type of individualized pair-wise centralized control may provide unsatisfactory results.
De-duplication can involve chunking and hashing an object. When there are two different computers involved in de-duplication, then both of the computers may perform chunking and hashing. The computers may perform the chunking and hashing and other de-duplication actions in different ways. When two computers intend to communicate de-duplicated data or intend to co-operate on a de-duplication task, then the fact that each of the computers may perform chunking and hashing and other de-duplication actions, in different ways can make it difficult, if even possible at all, for the two differently configured computers to work together.
In first generation de-duplication environments there may have been only a single de-duplication approach. Therefore all devices/processes that wanted to de-duplicate, communicate, and/or co-operate simply adhered to that one approach. However, as de-duplication systems matured, devices and processes become more varied and more sophisticated. Different de-duplication systems may have had multi-layered approaches to chunking and hashing and may have employed complex indexing systems that included primary and temporal indexes. The parsers may have become parameterized and customizable based on object context. Therefore it may have become ever more difficult to insure that the same deterministic approach to de-duplication would occur on both ends of a communication.